System and methods to support autonomous vehicles via environmental perception and sensor calibration and verification

ABSTRACT

An autonomous vehicle support system, including a lighting network ( 100 ) having, a plurality of light units ( 106 - 1, . . . , 106 -N) wherein at least one light unit includes at least one sensor type ( 110 - 1, . . . , 110 -N), and a centralized or distributed controller ( 102, 105 - 1, . . . , 105 -N), wherein a first light unit ( 106 - 1, . . . , 106 -N) receives sensor data from its sensor type ( 110 - 1, . . . , 110 -N), wherein the controller ( 105 - 1, . . . , 105 - 1 ) forms a local environmental perception of an area local to the first light unit ( 106 - 1, . . . , 106 -N), using the received sensor data from light unit, and receives sensor measurement data from an autonomous vehicle relating to at least a portion of the area, and cross-validates the local environmental perception of the area and the sensor measurement data from the autonomous vehicle.

The present invention is directed generally supporting autonomousvehicles by utilization of lighting networks with sensing andcommunication capabilities to (1) enhance environmental perception ofthe road by receiving support information from surrounding lightingnetwork infrastructure and (2) quantify and calibrate the vehicle'squality of perception of the surrounding environment, for example, thelocal area of near a light pole/unit. More particularly, variousinventive methods and apparatus disclosed herein relate to improvingautonomous vehicles performance by using information collected from thesurrounding Outdoor Lighting Network (OLN), as well as the vehicle'ssensor measurement data and calibration thereof. Thereby enablingdetection of inconsistency and failure of environment perception throughcomparison of vehicle's sensor measurement data with the surroundingOutdoor Lighting Network (OLN) local area sensor data to provide anenhanced understanding of road conditions. The OLN includes an array oflight units, and a network apparatus, hardware, and software formonitoring and managing the array, and analyzing sensor informationgathered from the array for targeted information communication.

One of the main services of the outdoor lighting is the improvement ofdrivers' perception of the road conditions and the surrounding area.Light serves as medium delivering signals from environment. Thesesignals are captured by eyes and further processed by human brain tocreate visualization of the surrounding world. However, in situationswhere cars can drive autonomously, people will not be continuously inthe control of the vehicles and they will not be required to observe theroad; instead they will spend the commuting time on various otheractivities.

The traditional scenario of people driving on a road will no longer bevalid. The fundamental goal of such outdoor lighting systems installedon roads and highways is to provide sufficient amount of light thatafter reflecting from a road delivers information captured by human eyesand once processed helps to understand the road conditions.

In a long-term scenario when people do not need to observe the roadduring commuting, the main use of the outdoor lighting will change fromhumans to systems in the car. So, there is a new role for lighting,which is not necessarily only to guide human vision, but also sensingsystems, and only dynamically adjusting lighting is needed in areas withpedestrian traffic—such as local roads and parking lots.

In the short to mid-term, the roadway scenarios will likely include bothautonomous cars and drivers, so during this transition period, there isan opportunity for the lighting system to provide new services tosupport the autonomous vehicles in addition to the traditionalillumination.

The technology of Self-driving, autonomous cars has been proven to work.Recent laws have been passed allowing cars to control themselves: in2012 self-driving cars were allowed in Nevada, California and Florida.Soon people will not be required to drive cars, but cars will drive bythemselves. This business paradigm shift in the demand of the type ofroad infrastructure support does not necessarily mean that outdoorlighting infrastructure will not be installed on the roads of thefuture. Instead, as vehicles continue to advance technologically,outdoor lighting will continue to evolve and have the opportunity toprovide enhanced services. Many of the current inventions that aim onimproving driving conditions for people will be shifted toward assistingautonomous vehicles control and maneuvering. Thus, the outdoor lightingnetwork (OLN) will serve a dual purpose: (1) it will support driving andsafety by communicating with vehicles, and (2), it will providecommunication and entertainment services to people who no longer areoccupied with steering vehicles.

Wireless sensing and actuating will still play critical role in OLNs;however, this time through interaction with the vehicles' controlsystem, thus closing the control loop, while minimizing the interactionwith people only to the entertainment content. Platforms for collectinginformation about road conditions through various sensors and cameraswill enhance the vehicles' perception of the road conditions. Furtherdata collected from connected devices, such as mobile devices, e.g.smart phones, smart watches, etc., will be used to enhance vehicles'far-distance understanding of the road conditions. While vehicles willbe instrumented with the minimum of technological equipment allowing acar to drive in most of conditions—the limit that comes from vehiclesspace size and payload weight constraints—OLN heterogeneousinfrastructure will provide sensing equipment that will not physicallyfit in a car. And then, as cars drive by themselves, OLN will take careof passengers by sending entertaining content directly to their phones,tablets and other mobile devices connected directly to cellularnetworks, for example.

With prior art of the autonomous vehicles, the technology requirescollection of a priori information about a road and surroundingenvironment, before a self-driving car can safely navigate on a road.For example, Google car is an example of autonomous vehicle whichreal-time sensory perception is further enhanced by additionalinformation about the road environment, i.e. before Google lets its carsto drive on a road, it sends conventionally driven car to map the roadand the surrounding area. Once this data is collected, at-runtime theself-driving car uses a beam laser that generates 3D map of thesurrounding environment. The laser observations are compared with thestatic information about the environment, and synthesized together toprovide high fidelity information used by the vehicle's control system.

This navigation approach assumes that the road environment is static.For example, the system assumes that side-walks are always in the sameplace and that the road has fixed width and known distance to thesurrounding building infrastructure. However, road conditions are oftendynamic, and obstacles may appear. Furthermore, environmental changestake long period of time to be detected, and therefore cannot beobserved at run-time by a vehicle. These slow changes require periodicre-scanning of the surrounding environment and bringing the 3D maps upto date.

Consequently, autonomous vehicles struggle with understanding theenvironment under such scenarios. While advancement in the vehicle'ssensor technology continues, it is unlikely that at any time a car byitself will have ability to fully observe and understand the dynamics ofthe road environment, i.e. a single car by itself will not be able to“see” behind physical objects nor predict actions from other drivers.So, for example, a person getting out of a bus and attempting to enterthe road to cross on the other side will still continue to beundetectable by the autonomous vehicles. Furthermore, an arriving car ora motor bike that enters a crossroad on a red light from another streetlocated behind a building will not be detectable by the autonomousvehicle itself, and thus the vehicle will not know that it needs to stopto prevent collision.

Moreover, autonomous vehicles make control decisions based on thereadings from the vehicle's sensory system. The challenge in theself-driving vehicles is in discovery of faults in the measurements ofthe sensor system and establishing how reliable the sensor measurementsare. This is a critical aspect of building a close-loop system, such asautonomous vehicle, because the actuation signals come from a controlsystem that computes decisions based on its perception of theenvironment. Thus, a car fails to navigate correctly when the sensormeasurements are incorrect.

The sensor system may fail due to multiple factors, such as long timeuse or car accidents. The first one can be solved by periodic carinspection and sensor replacement. Although period inspections help keepthe system running, inspections will only be done sporadically, andcannot detect malfunctions while the vehicle is being used, this is aserious problem. Malfunctions could include: problems with sensors,problem with vehicle lighting system (e.g. headlights alignment, missinglights, . . . ). Given the safety risks involved, there is a need for asystem to continuously monitor the accuracy and operation of the sensorsystem. The second type is an obvious accident that impairs car fromcontinuing driving.

The challenging sensor faults are the ones that occur when cars are inmotion and they cannot detect the faults, thus increasing the risk of anaccident. Such measurement errors can accumulate over time, as in caseof dust, or be interpreted by a car as an acceptable change to the roadconditions, e.g. in case of a bug hitting a sensor a vehicle maymistakenly detects a presence of a car that is not there ordust/snow/ice blocking the view of a sensor preventing it from detectinga car or obstacle nearby, which is also a serious issue.

Autonomous vehicles are also vulnerable to malicious tempering of theirsensors. Although, a vehicle can be instrumented with multiple sensorsto cross validate each other, it is still possible that multiple or allsensors fail due to malicious tempering. Thus a road-side infrastructureverifying the vehicle's perception is helpful to increase the drivingsecurity of the autonomous vehicles.

According to the principles of the present invention and to overcome orimprove the above limitations, we provide a system to detect groundtruth of road conditions and continuously monitor the [road] environmentand store the measurement data. The system includes light unitsinstrumented with sensors, a communication interface, a local databasefor storing measurement data, and a remote centralized database forcollecting road information for clusters of light units. Thecommunication interface allows transfer of messages to and from[autonomous] vehicles. The messages provide information about uniquechanges of the road environment, as well as to enable exchange ofmeasurements with [autonomous] vehicles and among each other tocross-validate their respective understanding of the road conditions.

In one embodiment, the present invention is a system comprising alighting network (LN) including an array of light units or luminaries,sensors and/or other integrated or connected devices (hereinafterreferred to as “light units”), a central management system (CMS), cloudserver or controller, a wired/wireless network, including software,firmware, for monitoring and managing the LN, as well as informationmanagement via the LN. The LN comprises multiple light units that mayoperate mainly in an independent mode where sensing, communication, andcontrol processes take place between the various light units. Furthercommunication and control may be provided between the light units and aCMS.

The controller, which may be a local control in a light unit or acentral management system (CMS) or a cloud server is operable to:receive and process light unit information, in particular, sensor unitdata or connected device information, collect and process sensor data,sensor measurement data from vehicles and connected device informationand detect events that are outside the perception range/view of thevehicles (events could also include combinations of several conditions(e.g. hazardous weather and poor visibility), for example, an event mayinclude: obstruction of vehicle visual perception (e.g. cameras/sensorshave impaired vision due to external factor), depending on the amount ofthe obstruction (e.g. as a percentage of the overall view), the systemcan set an alert/emergency to the driver, and/or to third party; anoutside object is detected (e.g. snow accumulation) over time, and aftera certain threshold an event is created to update/calibrate the vehiclemeasurement system); combine local processing at the light points and atthe remote cloud to analyze a time series of multiple sensormeasurements and compute trends in the environmental change and bycomparing the sensor measurements and the environmental changing trendswith the static information that vehicle has, to compute the differencein vehicle's perception; filter events and identifying the high priorityevents that complement the autonomous vehicles perception, i.e., eventsthat are not detectable or cannot be fully understood by the vehicleitself; receive and respond to vehicle queries; and broadcast alarmsabout events that reflect emergency conditions on a road that areoutside the vehicle's perception range; collect and process sensory datato create a profile containing a set of attributes that characterizesroad conditions; compute the difference between sensory ground truth andvehicles' measurements to assess the measurement error; cross-validatedand enhance the ground truth with third-party data, e.g. weatherforecast and traffic measurements can be used to validate measurements;detect emergency situation and safety issues based on the differencebetween the road condition's ground truth and the vehicle's perceptionof the road conditions and communicate an action to a vehicle includinga safe pull-over and guide further actions or the roadsideinfrastructure could disable the autonomous vehicle mode and return thevehicle to manual operation; warn a person present in a vehicle aboutthe vehicle's impaired perception based on the difference between theroad condition ground truth and the vehicle's perception of the roadconditions; and inform municipalities about malfunctioning vehicle (e.g.a vehicle with impaired visibility of the road conditions) and to warnother vehicles driving within a short distance from a malfunctioningvehicle; coordinate the operation of the identified lighting units as afunction of the lighting strategy, and send operation instructions toone more of light units to direct the identified light units to operatein accordance with the operation.

Another aspect of the invention provides a light unit in the LNconnected to a CMS, the light unit includes a processor; a memoryoperably connected to the processor; a sensing unit, and a communicationmodule operably connected to the processor for communication with theCMS and other light units. The sensor can be any sensor for sensing anyenvironmental condition. The processor is operable to: receive sensingdata and determine various conditions including lighting conditions,user/vehicle detection status, etc. with or without the CMS; transmitthe sensor data through the communication module to the CMS; receive anoperation instruction for operation of the light unit through thecommunication module from the CMS; and direct the light unit to operatein accordance with the operation instruction.

The foregoing and other features and advantages of the invention willbecome further apparent from the following detailed description of thepresently preferred embodiments, read in conjunction with theaccompanying drawings. The detailed description and drawings are merelyillustrative of the invention, rather than limiting the scope of theinvention being defined by the appended claims and equivalents thereof.

The following are descriptions of illustrative embodiments that whentaken in conjunction with the following drawings will demonstrate theabove noted features and advantages, as well as further ones. In thefollowing description, for purposes of explanation rather thanlimitation, illustrative details are set forth such as architecture,interfaces, techniques, element attributes, etc. However, it will beapparent to those of ordinary skill in the art that other embodimentsthat depart from these details would still be understood to be withinthe scope of the appended claims. Moreover, for the purpose of clarity,detailed descriptions of well-known devices, circuits, tools,techniques, and methods are omitted so as not to obscure the descriptionof the present system. It should be expressly understood that thedrawings are included for illustrative purposes and do not represent thescope of the present system. In the accompanying drawings, likereference numbers in different drawings may designate similar elements.Also, the drawing figures are not necessarily to scale, emphasis insteadgenerally being placed upon illustrating the principles of theinvention.

FIG. 1 is a schematic view of an lighting network (LN) in accordancewith embodiments of the present system;

FIG. 2a is perspective view of a lighting system in accordance withembodiments of the present system;

FIG. 2b is perspective view of a lighting system in accordance withembodiments of the present system;

FIG. 3 shows a flow diagram that illustrates a process in accordancewith embodiments of the present system.

Embodiments of the present system may interface with conventionallighting infrastructures such as urban walkway, street, and/or highwaylighting systems to control one or more portions of conventionallighting systems. It should also be understood that the sensors of thesensing unit can be any sensor for sensing any environmental condition,ranging from any electromagnetic signals to acoustic signals tobiological or chemical signals to other signals. Examples include an IRdetector, a camera, a motion detector, an ozone detector, a carbonmonoxide detector, other chemical detectors, a proximity detector, aphotovoltaic sensor, a photoconductive sensor, a photodiode, aphototransistor, a photo emissive sensor, a photo electromagneticsensor, a microwave receiver, a UV sensor, a magnetic sensor, a magnetoresistive sensor, a-Rd a position sensor, and a RF scanners to identifyMobile devices (e.g. Bluetooth, wifi, etc.).

FIG. 1 is a schematic view of a lighting network (LN) 100, a controller,central management system (CMS) or a cloud service 102 and aninformation resources server 112 (e.g. weather, traffic, publicsafety/security reports or other, for example news media or Internetavailable information), in accordance with embodiments of the presentsystem. Although FIG. 1 shows the elements of the lighting network (LN)100 as discrete elements, it is noted that two or more of the elementsmay be integrated into one or device. The lighting network (LN) 100includes a plurality of intelligent light units or luminaries (and/orelectrical devices) 106-1 through 106-N (generally 106), a plurality ofillumination sources 107-1 through 107-N, a plurality of controllers105-1 through 105-N, a plurality of transmission/receive (TX/Rx) units109-1 through 109-N, a plurality of sensors 110-1 through 110-N, one ormore [autonomous] vehicle interface apparatus 122, connected device(s)114 and a network/communication link 108 which, in accordance withembodiments of the present system, may operably couple two or more ofthe elements of the present system.

The vehicle interface apparatus 122 may include any number of securityauthorizations methods (including conventional security methods and onesdescribed further below). The vehicle interface apparatus 122 can beimplemented as a dedicated device or incorporated in another device. Thevehicle interface apparatus 122 can be implemented in a mobile phone,PDA, computer (e.g., laptop, tablet such as an iPad), and the vehicleitself, mobile GPS device, any intelligent device/machine, a sensingdevice or any other device accessible to a user. The vehicle interfaceapparatus may operate independently as an autonomous device without userinteraction. The vehicle interface apparatus 122, in one embodiment,responds to received external stimulus (e.g. sensor data from light unit106), to initiate an appropriate system responses.

The vehicle interface apparatus 122 communicates with the OLN, using anydesired technology, such as a cellular data communication protocol(e.g., GSM, CDMA, GPRS, EDGE, 3G, LTE, WiMAX,), DSRC or WiFi radio,ZigBee protocol operating on top of the IEEE 802.15.4 wireless standard,WiFi protocol under IEEE standard 802.11 (such as 802.11b/g/n),Bluetooth protocol, Bluetooth Low Energy protocol, visual lightcommunication (VLC), or the like.

When LN 100 is installed, the GPS coordinate information of each element(e.g. light units 106, connected devices 114 (light poles, sensors 110,traffic lights, etc.) in the system is generally recorded, and isavailable to CMS 102. All the elements are typically further placed ontoa map, therefore it is known to the CMS 102, for example, which trafficlight regulates the traffic leading to certain light units 106. Thisinformation can be included manually at commissioning or can be deducedusing the relative GPS coordinates and the geographical map with taggedstreets and traffic flows, available e.g. on OpenStreetMap. Associationsbetween light units 106 can then be stored in the memory of the CMS 102.

The connected device 114 can be any element in a smart city connectedinfrastructure that can provide information to help the light units 106tune its detection behavior to improve robustness. The connected deviceis any device that includes an interface apparatus to communicate withthe LN 100 via network 108. Any desired technology, such as a cellulardata communication protocol (e.g., GSM, CDMA, GPRS, EDGE, 3G, LTE,WiMAX,), DSRC or WiFi radio, ZigBee protocol operating on top of theIEEE 802.15.4 wireless standard, WiFi protocol under IEEE standard802.11 (such as 802.11b/g/n), Bluetooth protocol, Bluetooth Low Energyprotocol, or the like, can be used.

The connected devices 114 may include the following: connectedpedestrian or bicycle units to distinguish traffic types to enable thesystem to behave differently depending on the traffic type; variableconnected traffic signs to allow dynamically steering traffic flows,open/close lanes as needed, direct drivers in parking areas etc.;connected surveillance cameras; connected traffic management systems;connected (interactive) kiosks and advertising.

The CMS 102 may include one or more processors which may control theoverall operation of the lighting network (LN) 100. The CMS 102 may alsobe “distributed” (e.g. de-centralized in-network processing orhierarchical system, for example, the StarSense system where eachsegment controller controls a sub-set of light units). Moreover, theprocessing may be distributed between the CMS 102 and one or morecontrollers 105, described further below. The CMS 102 may also access toother information about the system and the environment, such asdate/time of the day, historic detection data, condition of theinfrastructure etc., for example, received via Resource Server 112.Moreover, the CMS 102 may request information from the resources server112 and may determine when to change system settings based on receivedinformation and/or history information (e.g., traffic light status,security data, pedestrian data or other so-called “connected” data meansavailable from the Internet, for example). The system may includestatistical and/or heuristic engines to fit data. LN 100 can use a citymanagement dashboard application such as the Philips CityTouch.Accordingly, the CMS 102 may communicate with, the light units 106, thesensors 110, to send and/or receive (via Tx/Rx units 109) variousinformation in accordance with embodiments of the present system.

The memory in the LN and CMS may include any suitable non-transitorymemory and is used to store information used by the system such asinformation related to operating code, applications, settings, history,user information, account information, weather related information,system configuration information, calculations based thereon, etc. Thememory may include one or more memories which may be located locally orremote from each other (e.g., a surface area network (SAN).

As noted above, the CMS 102 stores information in the memory (e.g.,historical information) which it receives and/or generates for furtheruse such as to determine lighting characteristics and sensor detectionthresholds in accordance with embodiments of the present system. As newinformation is received by the CMS 102, the stored information may thenbe updated by the CMS 102. The CMS 102 may include a plurality ofprocessors which may be located locally or remotely from each other andmay communicate with each other via the network 108.

The network 108 may include one or more networks and may enablecommunication between one or more of the CMS 102, the light units 106,the sensors 110, using any suitable transmission scheme such as a wiredand/or wireless communication schemes. Accordingly, the network 108 mayinclude one or more networks such as a wide area network (WAN), a localarea network (LAN), a telephony network, (e.g., 3G, a 4G, etc., codedivision multiple access (CDMA), global system for mobile (GSM) network,a plain old telephone service (POTs) network), a peer-to-peer (P2P)network, a wireless fidelity (WiFi™) network, a Bluetooth™ network, aproprietary network, the Internet, etc.

The Resource server 112, which may include other related informationresources such as proprietary and/or third party news media and Internetrelated resources which may provide information such as public safety,security, regulatory, traffic, weather, road condition reports and/orforecasts to the CMS 102 and/or the light units 106. This informationmay be used to further refine a light units 106 local or broadenvironmental perception of an area.

The sensors 110 may include a plurality of sensors types such as sensors110 which may generate sensor information based on the particular sensortype such as image information, status information (e.g., light unitoperative, non-operative, etc.), radar information (e.g., Dopplerinformation, etc.), geophysical information (e.g., geophysicalcoordinates obtained from, for example, a global positioning system(GPS)), pressure information, humidity information, etc. The sensors 110may be located at one or more geophysical locations or integrated into alight unit 106, and may report their location to the CMS 102. Eachsensor 110 may include a network address or other address which may beutilized to identify the sensor.

The light units 106 may include one or more illumination sources 107such as lamps (e.g., a gas lamp, etc.), light emitting diodes (LEDs),incandescent lamps, fluorescent lamps, etc., and may be controlled bythe controller 105. The illumination sources may be configured in amatrix (e.g., a 10×10 matrix of illumination sources) in whichillumination characteristics such as illumination pattern, intensity,spectrum (e.g., hue, color, etc.), polarization, frequency, etc., fromone or more of the plurality of illumination sources and/or lightpattern for a plurality of illumination sources, may be activelycontrolled by the system.

FIG. 2A is perspective view of the lighting system 100 (showing aportion of outdoor lighting network (LN) 100) in accordance withembodiments of the present system). The lighting system 100 may besimilar to the lighting network (LN) 100 and may include a plurality oflight units 106-1 through 106-N which may both illuminate an area orsurface (such as a street) and detect the presence of objects in adetection zone 207. One or more of the light units 106 may include oneor more of an illumination source 107, a controller 105, a Tx/Rx unit109 (not shown) and may also include connected devices 114 (not shown),illustratively a traffic light.

Light units 106 detect the presence of object/pedestrians/etc. in alocal area or detection zone 207. This information can be used formonitoring purposes and stored in the light units 106 memory or CMS 102memory for evaluation. Each light unit 106 creates a detection signalthat combines aspects of the sensor output signals useful to performdetection, and that presence is assessed by comparing such detectionsignal to a detection threshold. Thus, detection performances dependonly to the setting of the detection threshold, in this case: if thedetection signal is higher than the detection threshold, presence isdetected, otherwise not. It should be noted that this is anoversimplification, since presence detection algorithms are typicallysophisticated processing algorithms that use a large number of signalqualities to assess presence.

Light pole 106 instrumented with sensor types 110 (e.g. a laser, cameraand various sensors, such as motion and distance, as shown in FIG. 1),which together observe the environment and send this information toautonomous car(s) 205. For example, an infrared camera (not shown)detects motion from a vehicle that passed in front of a sensor. Thesensor data is stored in the light unit 106 memory, processed, andbroadcasted to nearby vehicles 205.

The lighting system 100 not only monitors the road conditions, but alsoobserves the surrounding environment. For example, to understand theenvironment dynamics, sensors monitor surrounding sidewalks, buildings,and nature, to bring awareness on possible adults and children enteringthe road from a sidewalk or a building, and animals that run out offorest onto the road.

The collected data may be stored in a light unit's 106 local memory. Atthe light unit 106, the data may be locally processed by controller 105(however, it is noted the CMS or remote cloud service 102 may alsopreform this processing) For example, the short-term data processingalgorithms detect patterns and suspicious or emergency activity thatrequires immediate attention. These emergency activities put real-timeconstrains on the vehicles, which must quickly react and adapt theirdriving to prevent collisions or accidents.

Illustratively, the controller 105, CMS or remote cloud service 102processes the sensor data to observe the (short and) long term changesin the road environment. For example, using measurements from multipleweeks, the cloud service notices differences in a shape of a tree. Or,using measurements from an hour, the cloud service detects snowaccumulated on the sides of the road. In each case, the cloud servicesends the computed results back to the light unit 106, which furtherbroadcasts the changes to nearby vehicles 205.

In emergency situations, the real-time constraints are mandatory toprovide application services. For example, the system must detect achild running into the street and because the child stands behind aparked car, the incoming vehicle cannot detect it, but once lightingsystem 100 detects a child on the street it must quickly inform thearriving vehicle 205, giving it enough time to stop. The real-timeconstraints are tighter when a possible collision can occur between twovehicles.

The processed sensor data returns a list of events that occur in thesurrounding environment. These events report the dynamic actions thatoccur on the streets, and the changes in the surrounding environmentthat are slow and accumulate over the time: events that are notdetectable by a car. The last type of events requires cloud server basedsupport, which combines a time series of multiple sensor measurementsand computes trends in the environmental change. Further, by comparingthe sensor measurements and environmental changing trends with thestatic information that vehicle has, we enable computation of thedifference in vehicle's perception with respect to lighting network 100ground truth perception.

Further, the lighting system 100 sorts the computed events based on apriority. Some events that lighting system 100 detects may not berequired to be sent to vehicles 205, but other events, which maycontradict with vehicle's 205 perception or may be critical to driversand pedestrians safety, are immediately sent to the autonomous vehicles'205 control system. Once the critical events are detected, the lightingsystem 100 computes the list of the surrounding vehicles 205 in motionthat should be immediately informed about the dynamic conditions.

Thus, in the manner described above, the system forms a (1) perceptionof the local area or environment surrounding a light unit 106 orconnected device 114 or (2) a perception of a broad area or environmentthat includes many local areas of light units 106 or connected devices114.

In another embodiment, the events are broadcasted by the lighting system100 to nearby vehicles 205, and each vehicle 205 incorporates thesereceived events in making its driving decisions. An event, for example,could be a dynamic action that occurs on the street. The light pointsthat detect this event alert the nearby vehicles 205 by sending theevent information including event description and geo-location (whichcan be calculated by each light point based on its own location).Further, when a vehicle 205 encounters some confusion regarding theenvironment (e.g., a pile of snow), it may send inquiries to nearbylight units 106, and the light units 106 will reply with the updateddata about the environment.

One of the advantages of the outdoor lighting system based roadmonitoring system is the data ownership. Currently, there are only asmall number of companies that collect high-quality images about roadsand surrounding areas. This data, which is critical to allow vehicles tonavigate, is privately owned. However, when a city purchases thelighting system 100, the data either belongs to the city or to thelighting system 100 provider (e.g. Philips). Another advantage is thatthe lighting system is permanent and will monitor the area for manyyears, without requirement specific intervention. Therefore, it canprovide a long term solution that enables the autonomous vehicle systemsto be calibrated. While there might be different trade-offs in the modelof data ownership between the two, because the data does not belong toany specific autonomous driving or automotive manufacture competitor,all kinds of self-driving vehicles will potentially be allowed to driveon the roads and have the same access to road information.

As shown in FIG. 2B each light pole is monitoring a fixed area of a road207. This area 207 could be adapted by adjusting the position of thesensors or by just processing the data. For instance, focusing a cameraon a specific area. This can be coordinated from the CMS 102. This couldalso be used to ensure the “view” of the lighting system is the same ofthe vehicles and ensure the calibration is successful. As a vehiclecontinues to travel on a road, it monitors its local area 209. Because avehicle observes the same area 211 (or at least a portion thereof) ofthe road as the nearby light units 106, the vehicle's perception of theroad should be similar to the light units' understanding of the roadconditions. Because a vehicle's sensors monitor the same area as lightunits' sensors, by comparing their observations, we can detect errors inthe vehicles' understanding of the road conditions.

To ensure road safety, vehicles must continuously monitor the quality oftheir sensory system. The sensor measurement validation mechanismrequires comparison of the vehicle's observation with a ground truth ofthe road conditions. Therefore, there exists a need of establishingground truth of the road conditions by a system other than vehiclesthemselves.

The lighting network 100 collects sensor data to establish ground truthabout the road conditions. As described above, each light unit 106 isinstrumented with a set of sensors 110, some of which are similar to theones used by vehicles 205, and enhanced with a sensor system that couldnot meet the power, size or weight constraints of cars. Because thelight units' 106 understanding of the road conditions can be faulty aswell, the lighting system 100 exchanges the measurements or data amongthe nearby sensors 110 to receive additional information about thesurrounding area and further enhance the value of the ground truth. Aphysical model or perception of the local area or surroundingenvironment is used to understand the relation among surrounding lightunits 106, i.e. traffic conditions might be similar between two nearbylight units 106.

To further enhance the understanding of the ground truth, the lightingsystem 100 cross-validates its understanding of the nearby roadconditions with third-party services using Connected device 114 orResource Server 112. For example, cell-phone based traffic estimationscan support cross-validation of the sensor measurements that detectpresence of vehicles on the road. City's planning information regardingscheduled road work or reports with issues about road conditions areused to localize points of reference between different sources of dataand to further improve the understanding of ground truth.

On the road, vehicles 205 and light units 106 exchange theirunderstanding of the road conditions. Each vehicle 205 starts withidentifying itself and its position on the road. The position ispresented as a GPS coordinate and timestamp. Further, the vehicle 205sends information about the road condition. The vehicle 205 reports itsdistance to the sides of the road and how far it can see the road ineach direction. Then, vehicle 205 reports objects that it believes to bein motion. With respect to its position, a vehicle reports location anddistance to other vehicles and as well as pedestrians. Finally, thevehicle 205 reports its location with respect to the surrounding staticinfrastructure, e.g. building, light poles, and trees.

Similarly, each light unit 106 beacons messages with information aboutthe conditions of the road being monitored by the particular light unit106. The beacon message consists of the light unit 106 location anddescription of the road: the road width, distance from a road to anystatic objects, and description of dynamic objects that are on the road.

After vehicles exchange their information, both compare their ownperception of the road conditions with another. On the vehicle side, itperiodically monitors its status of the sensory system and computes thesensors' trust value. This trust value allows a vehicle to estimate whensensors become faulty and when sensor calibration and online measurementverification is needed. On the lighting system 100 side, light units 106compare the vehicles' 205 sensor traces with their ground truthmeasurements. For each car, lighting system 100 computes the vehicle's205 trust value. The computed trust value is reported to the vehicle205, and it is used to estimate global quality of perception among allvehicles 205 driving on the monitored road way.

When a vehicle 205 detects errors in its sensor measurements, it eitherdecides to pull over, or continues to drive and compares itsmeasurements with other points of reference. Depending on the type of avehicle 205, car manufacturers may implement various strategies tohandle the inconsistency in road perception, or specific laws may bepassed to force detailed actions. When a light unit 106 detects errorsin vehicles' measurements it reports this event to municipalities andsends a warning message to other nearby vehicles 205. Further a lightunit 106 may send to the vehicle 205 with erroneous perceptionsuggestions with actions that would allow the vehicle to safely pullover and stop.

When lighting system 100 detects vehicles with faulty sensor system, itattempts to bring not only the awareness of the autonomous vehicles, butto also alarm the vehicle's passengers. As is known in the art, lightingsystem 100 can start or change its lights to bright warning colors, suchas red, and sends wireless information to nearby mobile devices, such assmart phones and tablets. Additional alarm sirens can be used to getattention of the malfunctioned vehicle's passengers.

Finally, the lighting based monitoring of the surrounding area forautonomous vehicles can be also applied to other technologies ofcommuting. For example, self-driving trains can use lighting system 100data gathered from the train stations to detect position of people whostand very close to the train tracks. Similarly, the lighting system 100can monitor the places where roads cross the train tracks, or areaswhere people attempt to walk across the tracks.

FIG. 3 shows a flow diagram that illustrates a process 300 in accordancewith embodiments of the present system. The process 300 may be performedby a system as shown in FIG. 1. The process 300 may include one of moreof the following steps. Further, one or more of these steps may becombined and/or separated into sub-steps, if desired. In operation, theprocess may start during step 301 and then proceed to step 303.

During step 303, the process determines if one of the sensor types 107detects new sensor data from a light unit 106 e.g. a detection of avehicle/user, or if new connected device 114 data is received. If thisdetermination is Yes, then the process proceeds to step 305.

During step 305 of the process, some or all of the data from sensors 226from each light unit 106 and/or connected device 114, which may includeinformation related to monitored areas in the vicinity of one or moreluminaires/connected devices in accordance with embodiments of thepresent system, is sent to CMS or cloud service 102 (or one or morecontrollers 105). After obtaining the information, the process maycontinue to step 307.

During step 307, the process analyzes the sensor data, by either theCMS/cloud service 102 or one or more controllers 105. For example, theprocess may analyze if detection in a respective light unit 106 is a“true” or “false” detection; establish ground truth of road conditionsand [road] environment; and forms an overview of the surroundingenvironment (e.g. objects, geography, weather & traffic conditions,etc.), as further described above.

In step 309, if the analyzed data should be broadcast or sent tovehicles 205, if not, the process proceeds to step 315. If yes, theprocess proceeds to step 311, and the analyzed data is sent, accordingthe embodiments present invention. After completing step 311, theprocess continues to step 313,

During step 313, the present system may form and/or update historyinformation (e.g., statistical information) of a memory of the presentsystem in accordance with the data, detection thresholds, number of“true” or “false” detections or other received information. For example,an indicator for behavior change, dimming schedule, ambient level, andother parameters, e.g. road type, traffic volume, weather status, thesensor information, day, date, time, user travel patterns, etc. whichinformation may be used at a later time. After completing step 313, theprocess may continue to step 315.

During step 315, the present system may determine whether to repeat oneor more steps of the process. Accordingly, if it is determined to repeatone or more steps of the process, the process may continue to step 303(or to another step which is desired to be repeated). Conversely, if itis determined not to repeat one or more steps of the process, theprocess may continue to step 317, where it ends. The process may berepeated at certain periodic and/or non-periodic time intervals. Byrepeating the process, history information may be accessed and used todetermine, for example, rate of change of the sensor information. Thisinformation may be used to determine and/or tune appropriate responsesin lighting system 100 to various situations and events.

While several inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto, inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

1. An autonomous vehicle support system, comprising: a lighting networkhaving: a plurality of light units wherein at least one light unitincludes at least one sensor type; and a centralized or distributedcontroller; wherein a first light unit receives sensor data from itssensor type; wherein the controller, forms a local environmentalperception of an area local to the first light unit, using the receivedsensor data from light unit, and receives sensor measurement data froman autonomous vehicle relating to at least a portion of the area, andcross-validates the local environmental perception of the at leastportion of the area and the sensor measurement data from the autonomousvehicle.
 2. The autonomous vehicle support system of claim 1, whereinthe controller forms a environmental perception, using received sensordata from first light unit and at least one of a second light unit,Connected devises and Resource server that obtains information fromadditional sources.
 3. The autonomous vehicle support system of claim 2,wherein the controller determines a ground truth relating to theperception by cross-validating the received sensor data used to form theenvironmental perception.
 4. The autonomous vehicle support system ofclaim 1, wherein the cross-validation includes comparing the lightunit's perception of the area and the sensor measurement data from theautonomous vehicle.
 5. The autonomous vehicle support system of claim 1,wherein the system transmits the validation information to theautonomous vehicle or broadcasts the validation information.
 6. Theautonomous vehicle support system of claim 1, wherein the controllerstores changes to the perception, when sensors sends new sensor data, toform a history.
 7. The autonomous vehicle support system of claim 6,wherein the controller generates a prioritized list of events relatingto the changes to the perception, and determines which events should besent to the autonomous vehicle or broadcast.
 8. A method of supportingan autonomous vehicle using a lighting network, the lighting networkhaving a plurality of light units wherein at least one light unitincludes at least one sensor type and a centralized or distributedcontroller; in communication with the light units, the method comprisingthe steps of: receiving, in a first light unit, sensor data from a firstsensor type; forming, in the controller, a local environmentalperception of an area local to the first light unit, using the receivedsensor data from light unit; receiving sensor measurement data from anautonomous vehicle relating to at least a portion of the area;cross-validating the local environmental perception of the at leastportion of the area and the sensor measurement data from the autonomousvehicle.
 9. The method of supporting an autonomous vehicle using alighting network of claim 8, further including the step of, forming, inthe controller, a broad environmental perception, using received sensordata from first light unit 106 and at least one of a second light unit,Connected devices and Resource server that obtains information fromadditional sources.
 10. The method of supporting an autonomous vehicleusing a lighting network of claim 9, further including the step of,determining, in the controller a ground truth relating to the perceptionby cross-validating the received sensor data used to form the broadenvironmental perception.
 11. The method of supporting an autonomousvehicle using a lighting network of claim 8, wherein the step ofcross-validating includes comparing the light unit's perception of thearea and the sensor measurement data from the autonomous vehicle. 12.The method of supporting an autonomous vehicle using a lighting networkof claim 8, further including the step of transmitting the validationinformation to the autonomous vehicle or broadcasting the validationinformation.
 13. The method of supporting an autonomous vehicle using alighting network of claim 8, further including the step, changing theperception, in the controller when sensor sends new sensor data, andforming a history.
 14. The method of supporting an autonomous vehicleusing a lighting network of claim 13, wherein the step of changing theperception includes generating a prioritized list of events relating tothe changes to the perception, and determining which events should besent to the autonomous vehicle or broadcast or sent to third parties.15. The method of supporting an autonomous vehicle using a lightingnetwork of claim 14, wherein the list of events include one or more oftraffic or weather conditions, detection of a road emergencysituation/safety issue, and detection of vehicle malfunctions.